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1.  Statement  of  the  Problem  Studied 

Techniques  had  been  developed  to  analyze  and  design  hybrid  optical  and  digital 
processing  imaging  systems  that  have  an  extended  depth  of  field.  However,  there  was  a 
need  for  analysis  and  design  tools  in  imaging  systems  that  attempt  to  reduce  the  depth  of 
field  or  provide  passive  ranging  by  range  gating  an  image.  Range  gating  of  an  image 
would  be  of  great  image  in  passive  imaging  systems  where  the  information  not  of  interest 
-  that  outside  of  the  plane  of  interest  -  would  not  corrupt  the  slice  of  the  image  of 
interest. 


2.  Summary  of  Most  Important  Results 

Research  during  the  period 

All  of  the  imaging  systems  considered  in  this  research  involved  optical  image 
acquisition  and  digital  signal  processing.  The  systems  were  jointly  designed,  so  that  the 
signal  processing  and  the  shape  of  a  phase  plate  to  be  placed  in  the  aperture  stop  were 
designed  so  that  the  impact  of  one  on  the  other  was  taken  into  consideration. 

The  work  early  during  the  period  of  this  grant  centered  on  a  technique  that  is 
analogous  to  range  detection  in  radar.  Previous  work  was  similar  to  range  estimation.  In 
the  previous  work,  multiple  objects  in  the  field  of  regard  caused  errors  in  the  range.  The 
work  on  range  detection  allowed  the  determination  of  the  ranges  of  several  objects  in  the 
field  or  regard.  This  was  done  by  modifying  the  optics  so  that  the  modulation  transfer 
function  (MTF)  had  a  peak  that  moved  with  the  distance  to  the  object.  This  work  used 
design  tools  that  work  in  the  spatial  frequency  domain.  The  validity  of  the  approach  was 
confirmed  in  both  simulations  and  experiments.  This  work  is  described  in  the  paper  with 
Johnson  and  Dowski. 

The  work  during  the  second  phase  of  the  report  period  concentrated  on  design 
tools  that  work  in  the  spatial  domain.  That  is,  the  design  was  on  the  point  spread  function 
(PSF),  not  the  MTF.  New  design  tools  were  developed,  and  a  new  metric  to  quantify  the 
blurring  of  the  defocused  image.  We  chose  the  angle  in  Hilbert  space  between  a 
defocused  PSF  and  the  in-focus  PSF.  This  metric  was  used  in  the  design  of  a  phase  plate 
to  maximize  the  differences  between  the  in-focus  and  out-of-focus  PSFs.  Simulations 
showed  that  the  rate  of  change  of  the  PSF  with  misfocus  was  doubled  in  comparison  with 
a  conventional  imaging  system.  This  work  is  described  in  the  paper  with  Sherif  Sherif, 
and  in  his  dissertation. 

In  the  latter  period,  an  information  theory  approach  was  used,  and  the 
orthogonality  of  the  images  was  considered.  This  work  led  to  special  forms  of  structured 
illumination  that  can  be  used  to  reduce  the  depth  of  field  of  an  imaging  system.  This 
work  is  described  in  the  attached  report. 
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Impact  on  the  Field 

An  indirect,  but  important,  result  of  the  research  during  this  period  was  an  impact 
on  the  growth  of  the  interest  in  the  field  of  hybrid  optical/digital  imaging  systems,  or 
“integrated  computational  imaging  systems”  or  “wavefront  coding”  as  it  is  also  called. 
Workshops  were  held  in  Boulder,  Colorado  (organized  by  David  Brady,  now  at  Duke 
Univ.)  and  in  North  Carolina  (Organized  by  Joe  Mait).  The  Optical  Society  of  America 
held  a  Topical  Meeting  on  the  subject  of  Integrated  Computational  Imaging  Systems  in 
Nov.  2001  at  Albuquerque,  New  Mexico.  In  May  2002,  there  was  a  special  issue  on 
Integrated  Computational  Sensors  for  SPIE’s  Optics  in  Information  Systems  Technical 
Group.  In  October  2002,  there  was  a  special  issue  of  Applied  Optics  on  the  subject.  In 
August  2003,  there  was  a  special  issue  of  SPIE’s  newsletter  Optics  in  Information 
Systems  on  Computational  Imaging.  Optics  Express  had  a  special  issue  on  Integrated 
Computational  Imaging  Systems  on  8  Sept  2003.  There  was  a  recent  DARPA  workshop 
and  a  Broad  Area  Announcement  on  the  topic  of  computational  imaging  systems. 
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Appendix  A 

Reduced  Depth  of  Field  Studies:  1/03-8/03 


Benjamin  Breaker,  Imaging  Systems  Laboratory 
Dept,  of  Electrical  and  Computer  Engineering 
Univ.  of  Colorado 
Boulder,  Colorado  80309-0425 


Introduction 

For  a  hybrid  imaging  system,  where  post-processing  algorithms  operate  with  custom 
optics,  the  desired  image  does  not  need  to  be  perfectly  reconstructed  on  the  surface  of  the 
sensor  array— but  the  information  content  of  the  image  does  need  to  be  preserved.  When 
this  concept  is  applied  to  a  reduced  depth  of  field,  we  can  understand  that  the  information 
of  the  desired  object  plane  must  be  preserved  (although  rearranged)  and  that  the 
information  from  all  other  planes  must  be  maximally  removable  from  the  information 
about  the  desired  object  plane.  Having  established  this  need  for  separability,  we  can  use 
the  orthogonality  of  the  signals  (or  intensity  distributions)  to  evaluate  how  separate  each 
interfering  signal  is  from  the  desired  signal.  This  aspect  of  the  study  established  limits 
regarding  signal  orthogonality  when  passed  through  linear  systems,  examined  the 
relationship  between  separation  distance  and  energy  spreading,  and  then  demonstrated 
how  structured  illumination  could  be  used  as  a  tradeoff  to  increase  the  energy  spreading 
of  undesired  signals. 


Figure  1  Schematic  of  an  image  formed  by  passing  light  through  a  single  system  from  two  objects  at 

different  distances. 
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Induced  Orthogonality 

We  explored  induced  orthogonality— increasing  signal  seperability  relative  to  other 
signals  within  the  system — in  a  linear  system,  as  illustrated  in  Fig.  1.  One  object  is  in  a 
position  to  be  imaged  in  the  focal  plane  and  another  is  in  a  more  distant  plane.  By 
exploring  the  standard  equations  and  treating  each  image  as  a  separate  input  to  the 
system,  we  have  shown  that,  for  a  linear  system  using  incoherent  light,  induced 
orthogonality  without  loss  of  the  desired  signal  is  bounded  by  the  correlation  between  the 
desired  signal  at  the  object  plane  and  the  interfering  signal  at  the  object  plane.  Once  the 
interference  signal  is  propagated  to  the  in  focus  object  plane,  the  two  signals  become 
inseparable — linearly  combining  as  a  single  signal.  Only  if  the  two  signals  were  spatially 
separable  (did  not  overlap  either  in  space  or  in  spatial  frequency)  within  the  object  plane 
can  the  signals  be  divided  and  extracted  by  signal  processing.  Many  times,  this  signal 
processing  is  dependent  on  the  human  vision  system  for  pattern  recognition. 

The  relationships  that  indicate  inseparable  signals  in  the  previous  paragraph  do  not 
preclude  the  removal  of  a  large  portion  of  the  interfering  signal  with  some  loss  to  the 
desired  signal,  but  that  aspect  of  has  yet  to  be  explored.  Any  solution  that  preserves  the 
information  content  of  a  desired  signal  while  dramatically  reducing  the  information  in 
another  plane  must  incorporate  processes  that  are  non-linear  with  respect  to  spatial 
frequencies  —such  as  a  high  numerical  aperture. 


Information  Content 

Once  it  was  understood  that  the  input  signal  separation  limits  performance,  it  became 
important  to  understand  how  the  information  content  of  signals  changes  with 
propagation.  This  will  allow  us  to  understand  the  transition  of  energy  from  the-out-of- 
focus  object  plane  to  the  in-focus  object  plane — a  phenomena  that  is  ultimately  the  only 
way  for  the  interfering  out-of-focus-signal  to  be  modified.  The  decrease  in  the  energy 
density  of  a  uniformly  illuminated  object  is  illustrated  Fig.  2.  As  the  undesired  object  is 
spatially  separated  from  the  desired  object,  the  signal  due  to  the  undesired  object  will 
change  according  to  this  plot. 
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Figure  2.  Out  of  focus  energy  as  a  function  of  the  misfocus  (in  mm).  The  structured  illumination 
(solid  line)  falls  off  faster  than  the  uniform  illumination  (dotted  line),  thereby  allowing  for  reduced 
depth  of  field. 


When  exploring  the  presence  of  information  and  our  ability  to  capture  that  information,  it 
is  important  to  keep  the  uncertainty  principle  in  mind  0.  For  a  given  piece  of  optical 
information,  we  can  know  its  spatial  and  temporal  properties  to  within  the  following 
constraints: 


AxAfxAyAfyAzAf2AtAv  s  1 

To  some  extent,  these  properties  may  be  traded— increasing  axial  resolution  (Az)  at  the 
expense  of  transverse  resolution  ( Ax  or  Ay ),  for  example.  System  properties  such  as  the 
numerical  aperture  enforce  limits  on  the  transverse  frequencies  ( A/x  and  A f  ),  while  the 

wave  nature  of  light  constrains  the  values  for  A fz ,  Ay ,  and  Ax .  Manipulations  of  the 

equation  result  in  a  tradeoff  between  the  classical  axial  resolution  and  the  temporal 
resolution  [8]. 


Az  2i 


{NAj  AtAv 

Although  this  concept  for  trading  information  is  interesting,  the  equation  says  nothing  as 
to  the  best  way  to  trade  the  information. 


Structured  Illumination 

Having  established  that  it  is  impossible  for  a  linear  imaging  system  to  increase  signal 
orthogonality,  as  described  in  the  previous  section,  we  began  to  search  for  ways  to  make 
the  signals  orthogonal  before  they  enter  the  imaging  system.  We  realized  that  the  only 
way  to  induce  orthogonality  onto  the  signal  was  through  controlled  illumination  and 
signal  recombination.  The  idea  explored  here  theorized  that  taking  n  images  with  the 
desired  object  illuminated  by  one  of  n  orthogonal  patterns  spanning  the  2-D  space  (such 
as  those  described  by  Chung  [1])  would  add  the  higher  spatial  frequencies  necessary  for 
energy  spreading  with  propagation.  As  long  as  the  illuminating  patterns  are  an  orthogonal 
spanning  set  of  the  2-D  signal  space,  the  signal  may  be  reconstructed  from  multiple 
images.  One  example  of  structured  illumination  is  shown  in  Fig.  3  for  a  random  phase 
illumination  while  the  energy  spreading  relative  to  a  uniform  illumination  is  compared  in 
Fig.  2.  Other  examples  of  structured  illumination  have  included  projected  (sinusoidal) 
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gratings,  and  follow  the  same  post-processing  method  of  additive  recombination  of  the 
images  illuminated  by  the  orthogonal  patterns  [2]. 


In  Focus  Ilium  In  Focus  Ilium  (phase) 


-5 

>  0 

5 


x  x 

Figure  3.  The  illumination  pattern  (in  the  in-focus  plane)  of  uniform  amplitude,  random  phase 
illumination  pattern. 

Future  studies  into  rotating  beam  patterns,  such  as  those  by  Piestun  [8],  as  the 
illumination  source  will  undoubtedly  add  additional  orthogonality— reducing  the  depth  of 
field.  A  system  could  be  created  where  the  object  is  illuminated  with  a  point-spread 
function  (PSF)  that  achieves  orthogonality  with  itself  through  propagation.  Since  the  PSF 
at  one  distance  may  be  spatially  orthogonal  (or,  at  least,  more  separable)  from  the  PSF  at 
another  distance,  the  energy  from  the  out  of  focus  plane  will  be  shifted  to  occupy 
different  spatial  regions.  This  means  that  an  image  of  the  object  not  in  the  focal  plane 
will  be  made  from  pixels  that  are  in  separate  locations  than  the  image  from  the  desired 
object.  With  knowledge  of  the  illuminating  beam,  the  energy  from  the  out  of  focus 
object  may  simply  be  removed  from  the  system.  By  building  up  the  image  as  a  linear 
sum  of  these  separable  illuminations,  the  image  could  be  constructed  with  a  reduced 
depth  of  field. 
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However,  such  goals  are  contradictory  in  a  normal  microscope. 
For  a  high  aperture  aplanatic  lens,  the  depth  of  lick!  is  1 1 1 


MAY  2opfr% 
VOL  13,  NO.  l  f 


' 

V  JSPIE’s 
international 
.Technical 


Newsletter 


Special  Issue: 

Integrated  Compu¬ 
tational  Sensors 

Edited  by  David  Brady, 

Duke  University 


Newsletter  now 

AVAILABLE  ON'LINE 

Technical  Group  members  are  being 
offered  the  option  of  receiving  the 
Optics  and  Information  Systems 
Newsletter  electronically.  An  e-mail 
is  being  sent  to  all  group  members 
with  advice  of  the  web  location  for 
this  issue,  and  asking  members  to 
choose  between  the  electronic  or 
printed  version  for  future  issues.  If 
you  are  a  member  and  have  not  yet 
received  this  message,  then  SPIE 
does  not  have  your  correct  e-mail 
address. 

To  receive  future  issues 
electronically  please  send  your  e- 
mail  address  to: 

spie-membership@spie . org 
with  the  word  OIS  in  the  subject  line 
of  the  message  and  the  words 
electronic  version  in  the  body 
of  the  message. 

If  you  prefer  to  receive  the 
newsletter  in  the  printed  format,  but 
want  to  send  your  correct  e-mail 
address  for  our  database,  include  the 
wordsprint  version  preferred 
in  the  body  of  your  message. 

http://spie.org/web/techgroups/ 

ois/pdfs/ 


Optics  in 

Information 

Systems 


Coding  the  wavefront  to 
extend  the  depth  of  field 


The  depth  of  field  of  an  imaging  system  can  be  increased  if  the  optics  are 
modified  and  some  signal  processing  is  done.1  Figure  1(a)  shows  an  ex¬ 
ample  where  the  objects  are  diatoms  viewed  under  J  OOx  optical  magnifi¬ 
cation  with  a  numerical  aperture  of  1.3.  Consequently,  the  depth  of  field 
of  the  system  is  very  small  as  seen  in  Figure  1(a).  After  coding  the 
wavefront  to  extend  the  depth  of  field  and  doing  some  signal  processing, 
the  image  of  Figure  1(b)  is  obtained.  This  shows  an  increase  in  the  depth 
of  field  by  about  an  order  of  magnitude.  How  is  this  possible?  We  use  a 
technique  that  is  very  unlikely  to  have  been  discovered  using  conven¬ 
tional  lens  design  techniques.  It  was  found  by  using  Woodward’s  ambi¬ 
guity  function  and  analogies  with  radar.  However,  after  a  means  of  cod¬ 
ing  the  wavefront  was  developed,  using  a  special  optical  element,  it  is 
possible  to  use  a  ray  trace  to  see  how  it  works. 

One  optical  element  that  can  extend  the  depth  of  field  by  coding  the 
wavefront  produces  an  optical  path  difference  that  varies  as  x*  +  y\  An 
element  such  as  this  is  placed  in  the  aperture  stop  of  the  imaging  system. 
After  this  modification,  the  rays  of  the  coded  wavefront  do  not  focus. 
They  are  spread  so  that  a  cross-section  of  the  rays  changes  very  little  with 
misfocus.  Figure  2  shows  the  two-dimensional  point  spread  functions  for 
a  normal  system  and  one  with  a  coded  wavefront.  Figure  2(a)  shows  an 
in-focus  point  spread  function  (PSF)  and  Figure  2(b)  shows  the  out-of- 
focus  PSF  for  a  normal  system.  Figures  2(c)  and  (d)  show  that  the  PSFs 
for  an  imaging  system  with  a  coded  wavefront  change  very  little  for  the 
same  misfocus.  The  PSFs  of  Figure  2(c)  and  (d)  cause  the  intermediate 
image  that  is  formed  with  the  modified  optics  to  appear  blurred.  This  is 
because  the  object  distribution  is  convolved  with  the  PSF  to  obtain  the 
image.  After  signal  processing  to  decode  the  intermediate  image,  the  PSF 
of  the  system  with  a  coded  wavefront  appears  as  sharp  as  the  one  of  Fig¬ 
ure  2(a).  Consequently,  it  is  possible  to  obtain  an  image  such  as  the  one 
shown  in  Figure  1(a)  by  modifying  the  optics  and  performing  signal  pro¬ 
cessing. 

What  is  the  cost?  In  addition  to  the  increased  signal  processing,  there 
is  a  reduction  in  the  signal-to-noise  ratio  (S/N)  in  the  final  image.  From 
the  point  of  view  of  a  3D  modulation  transfer  function  (MTF),  there  is  a 
fixed  amount  of  signal  that  can  either  be  concentrated  in  the  normal  focal 
plane,  or  spread  over  a  region  about  that  plane.2'4  If  spread  about,  the  level 
of  the  MTF  in  the  focal  plane  must  drop.  Hence,  there  is  a  loss  of  S/N  in 
the  mid  range  of  spatial  frequencies,  compared  to  the  case  of  in-focus 
images  with  a  conventional  imaging  system.  Other  forms  of  phase  plates 

continued  on  p.  8 


Figure  1.  Diatoms  imaged  with 
lOOx  with  a  numerical  aperture 
of  1.3.  (a)  with  a  conventional 
imaging  system  and  (b)  with  an 
imaging  system  with  a  coded 
wavefront. 
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Coding  the  wavefront 

continued  from  cover 

to  do  the  coding  cause  a  smaller  loss  in 
S/N,  but  there  is  always  some  loss, 
which  is  dependent  on  the  amount  by 
which  the  depth  of  field  is  increased. 
The  effect  of  this  loss  is  highly  depen¬ 
dent  on  the  dynamic  range  of  the  cam¬ 
era  being  used. 
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Figure  2.  Point  spread  functions  for  (a)  conventional  imaging 
system  in  focus,  (b)  conventional  imaging  system  out  of 
focus,  (c)  coded  imaging  system  in  focus  and  (d)  coded 
imaging  system  out  of  focus. 
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cal  power  (lens)  and  establish  weighted  fan-out 
interconnections.  Optical  outputs  broadcast  over 
these  weighted  interconnections  are  modulated 
(temporally  varied)  in  intensity  by  each  individual 
VCSEL  element  and  its  associated  Si  driver  cir¬ 
cuit. 

For  adaptive  vision  sensors,  the  design  of  the 
individual  Si  VLSI  chips  and,  in  particular,  the 
use  of  spatio-temporal  multiplexing  techniques  for 
network  implementation  and  signal  processing 
functions,  are  motivated  by  recent  advances.  Spe¬ 
cifically,  several  promising  biologically-inspired 
vision  algorithms  have  been  developed  that  can 
potentially  be  mapped  into  the  emerging  3D 
PMCM  platform. 

Assemblies  of  PMCMs  can  be  configured  for 
a  wide  variety  of  computational  tasks,  depending 
on  the  type  of  pre-detection  optics  employed.  For 
example,  in  an  architecture  designed  for  the  co¬ 
extraction  of  spatial  and  spectral  features  of  an 
image,  a  tunable  Fabry-Perot  etalon  or  liquid  crys¬ 
tal  filter  is  used  to  transmit  temporally-multiplexed 
spectral  ranges.  The  device  is  oriented  such  that 
the  complementary  spectral  information  is  re¬ 


flected,  as  shown  in  Figure  2.  This  configuration 
provides  for  efficient  usage  of  incoming  photons, 
and  at  the  same  time  provides  separate  process¬ 
ing  paths  for  color-based  and  gray-scale  features. 
The  spectral  range  can  be  altered  by  means  of  a 
variable  iris  diaphragm,  as  shown. 

To  date,  all  of  the  individual  components 
shown  in  Figure  1  have  been  successfully  de¬ 
signed,  fabricated,  and  tested.  In  particular,  both 
fan-out  and  fan-in  interconnections  have  been 
successfully  demonstrated.  Current  efforts  are 
focused  on  multichip-module  integration  and 
alignment  in  order  to  demonstrate  full  PMCM 
functionality. 
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